Gesture Recognition with Keypoint and Radar Stream Fusion for Automated Vehicles

We present a joint camera and radar approach to enable autonomous vehicles understand react human gestures in everyday traffic. Initially, we process the data with PointNet followed by spatio-temporal multilayer perceptron (stMLP). Independently, body pose is extracted from frame processed separate stMLP network. propose fusion neural network for both modalities, including an auxiliary loss each modality. In our experiments collected dataset, show advantages of gesture recognition two modalities. Motivated adverse weather conditions, also demonstrate promising performance when one sensors lacks functionality.